RuArg-2022: Argument Mining Evaluation

Argumentation analysis is a field of computational linguistics that studies methods for extracting arguments from texts and the relationships between them, as well as building argumentation structure of texts. This paper is a report of the organizers on the first competition of argumentation analysis systems dealing with Russian language texts within the framework of the Dialogue conference. During the competition, the participants were offered two tasks: stance detection and argument classification. A corpus containing 9,550 sentences (comments on social media posts) on three topics related to the COVID-19 pandemic (vaccination, quarantine, and wearing masks) was prepared, annotated, and used for training and testing. The system that won the first place in both tasks used the NLI (Natural Language Inference) variant of the BERT architecture, automatic translation into English to apply a specialized BERT model, retrained on Twitter posts discussing COVID-19, as well as additional masking of target entities. This system showed the following results: for the stance detection task an F1-score of 0.6968, for the argument classification task an F1-score of 0.7404. We hope that the prepared dataset and baselines will help to foster further research on argument mining for the Russian language. © 2022 ABBYY PRODUCTION LLC. All rights reserved.

Analyzing COVID-related Stance and Arguments using BERT-based Natural Language Inference

In this paper we present our approach for stance detection and premise classification from COVID-related messages developed for the RuArg-2022 evaluation. The methods are based on so-called NLI-setting (natural language inference) of BERT-based text classification (Sun et al., 2019), when the input of a model includes two sentences: a target sentence and a conclusion (for example, positive to masks). We also use translating Russian messages to English, which allows us to leverage COVID-trained BERT model. Besides, we use additional marking techniques of targeted entities. Our approach achieved the best results on both RuArg-2022 tasks. We also studied the contribution of marking techniques across datasets, tasks, models and languages of RuArg evaluation. We found that "<A:ASPECT> keyword </A:ASPECT>” gave the highest average increase over corresponding basic methods. © 2022 ABBYY PRODUCTION LLC. All rights reserved.

Extracting sentiments towards COVID-19 aspects

In this paper, we introduce a specialized Russian dataset and study approaches for aspect-based sentiment analysis of Russian users’ comments about the COVID-19. We solve two tasks, namely Relevance Determination (RD), which aims to predict whether a sentence is relevant to an aspect of the pandemic, and Sentiment Classification (SC), which classifies the sentiment expressed towards an aspect in a sentence. We applied and tested various methods of machine learning, including finetuning of the pre-trained RuBERT model. The best results in both tasks were obtained by RuBERT model in the Natural Language Inference (NLI) formulation. Copyright © 2021 for this paper by its authors.